Skip to content

MILAB-6648: Project template - #1767

Open
AStaroverov wants to merge 16 commits into
mainfrom
MILAB-6648_block-kind-prototype
Open

MILAB-6648: Project template#1767
AStaroverov wants to merge 16 commits into
mainfrom
MILAB-6648_block-kind-prototype

Conversation

@AStaroverov

@AStaroverov AStaroverov commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Greptile Summary

This PR introduces block kinds and project templates across the SDK, middle layer, registry, and block-authoring toolchain.

  • Adds typed block-kind descriptors, references, selectors, manifests, publication, registry projection, and implementation resolution.
  • Adds template-v1 parsing, validation, serialization, export, resolution, and application.
  • Threads template initialization parameters through block storage creation and adds template-parameter export from block models.
  • Adds kind build targets, structure rules, package scaffolding, example-block migrations, and integration coverage.
  • Important touched terms:
    • Block kind — a separately versioned typed initialization contract implemented by one or more blocks; this PR adds its descriptor API, package format, build/publication lifecycle, registry representation, and resolver.
    • Compiled block kind — the frozen runtime descriptor produced by defineBlockKind; it now carries the kind identity, schema version, parameter type, and template-parameter parser.
    • Block kind reference — the canonical {package-name}@{version} identity stored in block models and manifests; this PR adds formatting and parsing helpers.
    • Kind selector — an exact, patch-floating, or minor-floating version requirement used by templates; this PR defines selector parsing and range semantics, including pre-1.0 behavior.
    • Kind overview — the registry projection connecting kind versions to implementing block versions and publication channels; this PR adds reconciliation and resolution support.
    • Project template — a portable template-v1 document describing blocks, initialization parameters, and references; this PR adds its schema and import/export lifecycle.
    • Template entry — one block declaration within a project template; this PR adds validation, kind resolution, reference rewriting, and construction of the corresponding project block.
    • Template parameters — kind-validated values used to initialize block storage and regenerated during export; this PR adds both initialization and reverse-projection paths.
    • BlockPackProvider — the middle-layer abstraction used to resolve template kind selectors into concrete block packs; this PR adds registry-backed provider behavior.
    • DataModelBuilder — the SDK builder for versioned block storage; this PR associates it with a kind and passes kind parameters into its initialization factory.

Confidence Score: 5/5

The PR appears safe to merge.

The previously reported stale kind association, pre-1.0 selector range, and missing initialization-parameter paths are addressed in the current code, and no blocking failure remains.

Important Files Changed

Filename Overview
lib/model/common/src/template/project_template_v1.ts Defines the shared template-v1 document contract used by template parsing, validation, import, and export.
lib/node/pl-middle-layer/src/model/template_apply.ts Applies validated and resolved template entries to a fresh project while mapping template-local IDs to created blocks.
lib/node/pl-middle-layer/src/model/template_export.ts Exports project blocks and their kind-specific parameters into a portable template document.
sdk/model/src/block_migrations.ts Associates data models with block kinds and provides distinct default and parameterized storage-initialization paths.
tools/block-tools/src/v2/registry/registry.ts Reconciles kind implementation projections, including removing associations from a block's previously declared kind.
tools/block-tools/src/v2/registry/kind_resolver.ts Resolves exact and floating kind selectors using explicit kind-version range semantics.
tools/block-tools/src/v2/publish-block.ts Coordinates kind validation and publication before publishing the implementing block package.
sdk/block-kind/src/index.ts Exposes the block-kind authoring API and compiled descriptor construction.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  K[Block kind package] --> KB[Kind build and manifest]
  KB --> PUB[Kind-first publication]
  PUB --> REG[Registry kind overview]
  T[template-v1 document] --> PARSE[Parse and validate]
  PARSE --> RESOLVE[Resolve kind selectors]
  REG --> RESOLVE
  RESOLVE --> APPLY[Create project blocks]
  APPLY --> INIT[Initialize storage from params]
  INIT --> P[Project]
  P --> EXPORT[Export template params]
  EXPORT --> T
Loading

Reviews (2): Last reviewed commit: "refactor: enhance kind overview handling..." | Re-trigger Greptile

Context used:

@notion-workspace

Copy link
Copy Markdown

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

Comment thread tools/block-tools/src/v2/registry/registry.ts Outdated
Comment thread tools/block-tools/src/v2/registry/kind_resolver.ts Outdated
Comment thread sdk/model/src/block_migrations.ts
@changeset-bot

changeset-bot Bot commented Jul 29, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 2677325

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 29 packages
Name Type
@platforma-sdk/block-kind Minor
@milaboratories/pl-model-common Minor
@milaboratories/pl-model-middle-layer Minor
@milaboratories/pl-middle-layer Minor
@platforma-sdk/block-tools Minor
@milaboratories/ts-builder Minor
@platforma-sdk/model Minor
@milaboratories/milaboratories.monetization-test Patch
@milaboratories/milaboratories.pool-explorer Patch
@milaboratories/milaboratories.ui-examples Patch
@milaboratories/pf-driver Patch
@milaboratories/pf-spec Patch
@milaboratories/pf-spec-driver Patch
@milaboratories/columns-collection-driver Patch
@milaboratories/pl-client Patch
@milaboratories/pl-drivers Patch
@milaboratories/pl-mcp-server Major
@milaboratories/pl-deployments Patch
@platforma-open/milaboratories.software-ptabler.schema Patch
@platforma-sdk/ui-vue Minor
@platforma-sdk/pl-cli Patch
@platforma-sdk/test Minor
@milaboratories/pl-model-backend Patch
@milaboratories/pl-errors Patch
@milaboratories/pl-tree Patch
@milaboratories/ptabler-expression-js Patch
@milaboratories/uikit Patch
@platforma-sdk/tengo-builder Patch
@platforma-sdk/bootstrap Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.48018% with 75 lines in your changes missing coverage. Please review.
✅ Project coverage is 53.13%. Comparing base (7e54fd7) to head (9e79ae2).
⚠️ Report is 35 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...e/pl-middle-layer/src/middle_layer/middle_layer.ts 0.00% 33 Missing ⚠️
...b/node/pl-middle-layer/src/model/project_helper.ts 57.14% 13 Missing and 2 partials ⚠️
lib/model/common/src/drivers/pframe/spec/ids.ts 82.92% 3 Missing and 4 partials ⚠️
...ode/pl-middle-layer/src/block_registry/registry.ts 0.00% 6 Missing ⚠️
.../node/pl-middle-layer/src/model/template_parser.ts 82.60% 1 Missing and 3 partials ⚠️
...b/model/common/src/template/project_template_v1.ts 97.18% 1 Missing and 1 partial ⚠️
lib/model/common/src/template/template_form.ts 90.00% 0 Missing and 2 partials ⚠️
lib/node/pl-middle-layer/src/model/template_ids.ts 90.47% 1 Missing and 1 partial ⚠️
...node/pl-middle-layer/src/model/template_resolve.ts 96.49% 1 Missing and 1 partial ⚠️
...ddle-layer/src/block_registry/template_provider.ts 95.45% 0 Missing and 1 partial ⚠️
... and 1 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1767      +/-   ##
==========================================
- Coverage   53.76%   53.13%   -0.64%     
==========================================
  Files         366      411      +45     
  Lines       19679    21280    +1601     
  Branches     4341     4737     +396     
==========================================
+ Hits        10581    11307     +726     
- Misses       7795     8617     +822     
- Partials     1303     1356      +53     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@AStaroverov AStaroverov changed the title [DRAFT] MILAB-6648: block kind prototype MILAB-6648: Project templates Aug 7, 2026
@AStaroverov AStaroverov changed the title MILAB-6648: Project templates MILAB-6648: Project template Aug 7, 2026
@AStaroverov

Copy link
Copy Markdown
Collaborator Author

@greptileai

@AStaroverov
AStaroverov force-pushed the MILAB-6648_block-kind-prototype branch from 9e79ae2 to 866d82b Compare August 10, 2026 14:29
Mechanical, and a prerequisite rather than part of the feature: a kind is a
mandatory component of a V3 block, so the workspace cannot hold V2 example
blocks once the following commits land.

The `-v3` suffixed duplicates that existed for migration testing are removed
and their non-suffixed counterparts take their place, so every reference to
`enter-numbers-v3` / `sum-numbers-v3` in tests moves with them. `ml-legacy.test.ts`
and `v2.test.ts` covered the V2 path only and are dropped; the two cases from
`ml-legacy.test.ts` still worth keeping are carried into `ml-v3.test.ts`.

Each block's model also projects its params, which only compiles once the
params contract lands — see the sdk/model commit later in this branch.
`BlockKindReference` is the `{name}@{version}` string a published block advertises,
and `formatKindRef` is the one place it is built from a compiled kind. The reference
sits on the config container beside `code`, not inside a render envelope: which kind
a block implements is orthogonal to how it renders, and the container is what a
reader already has in hand.

The read side stays optional. Blocks published before kinds existed carry no
reference, and the middle layer must keep reading them.
A PColumn id carries the block id of the block that produced it, sometimes several
levels down: under wrappers, in every hop of a discovered path, and in the keys of a
qualifications map. Template export and apply both have to rewrite those ids, so the
walk lives beside the id codec rather than in either caller.

Rewriting is by value: an id that does not change comes back as the very same string
rather than a re-serialized equivalent, so a caller can tell "nothing to do" from
"rewritten to the same thing".
The schema a template file is parsed against, and the codec that converts the
references inside an entry's params between the two forms they take: a live `PlRef`
naming a block, and the two-key mapping a file uses to name another entry.

Two decisions the schema makes, both load-bearing:

  - `kind` is required on every entry, because the kind carries the params contract —
    an entry without one describes params nothing can check.
  - a kind selector is its own three-tier grammar (exact, patch-float, minor-float)
    rather than an npm range, so that below 1.0.0 the tiers stay distinguishable,
    where npm's `~` and `^` collapse into the same thing.

An entry may pin a version or pin a place, never both: a pinned place is what makes a
template resolvable without a registry, and a pinned version is what makes it
portable, so a file that claims both has said nothing definite.
A kind is a block's identity plus its init-params contract, declared in its own tiny
package so that two blocks can implement the same kind and a template can name one
without naming an implementation.

The params type is recovered from the declaration (`InferBlockParams`), and a kind
with wider params is deliberately not assignable to one with narrower params — a block
must not widen the contract it claims to implement.

`ts-builder` gets the rolldown and tsconfig pair a kind package builds with; a kind
ships as its own artifact because a block bundles its dependencies once and the kind
has to be readable without unpacking the block.
The structurer gains `kind/` as a fourth component beside model, workflow and ui, so
`block-tools` scaffolds it, validates its package.json, and refuses a block without
one. New blocks get a params parser stub that deliberately does not build: a kind whose
contract was never written should fail loudly at build time, not silently accept
anything.

A kind is built twice — once as its own publishable artifact, once bundled into the
block — because the registry must be able to read a kind's contract without unpacking
any implementation, while the block still needs it at runtime.

`checkKindVersionMatch` is the gate publishing runs before anything is written: the
kind a block claims and the kind that was built have to agree on name, org and version,
with an incidental npm-scope difference normalized away.
Publishing is kind-first: the version-match gate, then the kind, then the block. A
kind version's content folder is immutable — republishing identical content is an
idempotent no-op, republishing the same version with different content is a hard
failure, because a template that pinned that version would otherwise silently mean
something else.

Reconcile derives a per-kind overview (which kind versions exist, which blocks
implement each, in which channels) in the same pass that already reconciles packages,
rather than in a second reconciler. The overview is a derived view, so a channel change
after the fact re-derives it.

`resolveKind` maps a selector tier to an explicit semver range and picks the newest
matching kind version with a stable implementer, falling back to the any channel only
when the caller allows unstable.

The block-repo tests carry this on the same axes block publishing is already covered
on, and across two kind versions — with a single version on the registry every
selector picks the same block whether or not the ranges are right.
Three things arrive together because the type parameter that carries the kind's params
threads through all of them:

  - `create()` now takes a kind, and the authoring API can no longer produce a
    kind-less block. The kind handed to the data model and the one handed to `create()`
    are cross-checked, so there is one source rather than a precedence order.
  - `templateParams()` is required. It is the inverse of the data model's `init`:
    `init` builds data from params, this recovers the params that would rebuild the
    current data. A block whose state carries nothing worth restoring returns `{}` and
    says so, rather than exporting an entry that silently applies as a default block.
  - `parseTemplateParams` is read off the compiled kind, never declared per block.
    The contract belongs to the kind, so two blocks implementing it cannot disagree
    about what a valid params object is.

There is no unchecked path: what flows on to `init` is the parser's output, not its
input, so params the kind rejects never reach a block.

A known hole, left explicit in the tests: an extra field the kind does not declare is
not rejected, because a kind's params are a TypeScript type with no runtime schema of
their own beyond what the parser chooses to check.
Generated, and uniform by construction: one `kind/` package per example block, four
config files and a params declaration each. Nothing here is a design decision — the
contracts these kinds declare are whatever the corresponding block model already
projects in its `templateParams()`.

Worth skimming rather than reading, with one exception: the params type in each
`kind/src/index.ts` is the contract a template file is checked against, so it is the
one line per block that carries meaning.
Generated. `pnpm-workspace.yaml` picks up the `kind/` glob and the lockfile follows.
A dependency-order walk over the project structure, then a serializer that renders the
result as `template-v1` YAML. Structure order is already instantiation order, so the
walk sorts nothing.

All or nothing: no partial YAML is ever produced. The walk collects every problem and
the serializer adds its own, and they are reported together in one pass, because a
half-written template is worse than none.

The central check is that no live block id escapes into the file. A `PlRef` the codec
failed to rewrite is caught, and so is one hidden inside a string (the `EnrichmentRef`
case) or double-stringified. A column id naming a block the template describes is a
wire and is fine; one naming a block it does not describe is a fault. A block cannot
name itself, since its own id is only published once it has been written.

`mutator/project.ts` gains the kind reference on `BlockInfo` — `extractConfig`
normalizes the render envelope one level below where the kind sits, so it has to be
read off the container during the load that is already happening. The `initialStorage`
seam added here is what the apply path later fills in.
The three steps before anything is created: parse the YAML, check the document on its
own, then resolve each entry to a concrete block pack.

Parse errors are located the way the file is written, and every schema problem is
listed rather than just the first, because a hand-edited template usually has more than
one thing wrong with it.

Validation runs before creation, not during: forward references, self-references,
references to ids the file does not define, and — the case that only shows up with a
file from another machine — params still carrying a live block id from the project they
were exported from, including one hidden inside a string.

Resolution has two routes. An entry that names a place is read from that place and
consults no registry at all, which is the whole point of naming one; its declared kind
is then verified against what was actually found there. An entry that names a kind goes
to the configured registries in order, first hit winning, and when they disagree the
failure that got furthest is the one reported — a registry that has never heard of the
kind is less informative than one that has it but only unstable.

Every entry is attempted before anything is reported, so a document that half resolves
says so about both halves.
Creation, in file order, with an id map recording which block each entry became so the
next entry's references can point at blocks that already exist. A forward reference is
impossible by construction here rather than merely rejected.

Every entry goes through the params path, including one with no params, which is
applied as `{}`. There is no second, unchecked way in — that is what makes "the kind
checked these params" true of every block a template creates.

Params are turned into storage in the model VM before the transaction is touched, so a
kind rejecting them is a value the caller can report rather than a half-applied
project. Apply stops at the first entry it cannot add and reports what already landed:
partial is visible, never silent. A rejected entry does not become a reference target.

`MiddleLayer` gains both public entry points. They take a project id rather than an
open project, because exporting and applying are properties of the stored project, not
of a session with it — the export command lives on a project card where the project is
usually closed, and the import flow has just created the project and has no session yet.
Export and apply are inverses. The two halves only a live project can exercise are
deriving every block's params in the model VM off stored state, and resolving,
installing and placing blocks against a real backend; everything between them is unit
tested in the middle layer, so what is proven here is that the two ends compose.

Applying goes through the YAML text rather than the in-memory document, so what a user
would actually save is what the importer reads. Equivalence is stated up to renaming,
since applying creates new blocks with new ids.

The fidelity boundary is asserted rather than avoided: a field the kind declares
survives, a field it does not is lost and the test says so. Transport is held constant
— both blocks are installed from a folder on disk, with `resolveKind` stubbed to throw,
so a located entry reaching a registry would fail the test rather than pass quietly.
The working documents this branch was built from: the kind and lifecycle design, the
export and import designs, and the resolved-question log each carries. Kept because the
decisions behind several non-obvious choices — the selector grammar below 1.0.0, kind
built twice, params checked by the kind rather than the block — are recorded here and
nowhere else.

Separated into its own commit so the code can be reviewed without it, and so it can be
dropped or relocated without touching anything else.

@dbolotin dbolotin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So yes, the issue with the reference mapping is real (see #1781), and overall by-shape recognition seems to be not the best approach here.

* so a parser must not treat a specific id as meaningful.
*/
readonly parseTemplateParams: (value: unknown) => BlockParams;
readonly __PHANTOM_BLOCK_PARAMS__?: (p: BlockParams) => void;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need that, exactly? Not clear from the class docs.

export type { CompiledBlockKind, InferBlockParams } from "./descriptor";

/** A kind's identity, as declared in its own `package.json`, plus optional behaviour. */
export interface BlockKindMeta<BlockParams = unknown> {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this one if we have CompiledBlockKindV1 ?

Comment thread sdk/block-kind/src/index.ts
/** Reference to the block kind this data model belongs to, if declared. */
kindRef?: BlockKindReference;
/** The kind's runtime params check, threaded with {@link kindRef}. */
parseTemplateParams?: (value: unknown) => unknown;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parseInitializationParams

* Runtime check for params that did not come from a typed caller. Required — see
* {@link CompiledBlockKind}'s `parseTemplateParams` for what it must do and why it exists.
*/
parseTemplateParams: (value: unknown) => BlockParams;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parseInitializationParams, template is one way how we can initialize the block, another one will be tests, another MCP/AI.
Another thing here, let's not allow this function to alter parameters (like zod normalization), this logic should be on the block side. In other words if we do so, there will be a temptation to for example to fill in defaults of a certain parameters with a specific values here, and this will create an incorrect logic placement, any logic belongs to the block model code, and should change with block.


return {
addBlock: (request): AddBlockOutcome => {
const prepared = entries.get(request.id);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit "cheating", that is the whole point of isolating APIs, this function should receive all the information from the caller.

// A kind ships a runtime params check, so it is not a types-only package. zod is
// the default the scaffolded parser is written against; an author who validates by
// hand can drop this, which is why it is seeded rather than asserted on refresh.
zod: "catalog:",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

☣️

* cannot drift from the contract by being left behind. zod is the default; any function
* from `unknown` to `BlockParams` satisfies the slot, including a hand-written one.
*/
const Params = z.object({}).strict();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's get rid of zod?

* existing content is discarded rather than filtered.
*/
type KindTouchAccumulator = {
touched: Set<string> | null;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

touchedImplemeters/touchedBlocks?

Comment on lines +220 to +228
// Seed every existing kind overview empty so kinds orphaned by
// migration/removal are rewritten (or deleted) this pass. The block scan
// above already re-enumerates every live kind ref (refs live inside block
// manifests); this LIST exists solely to reset orphans.
const kindPaths = await this.storage.listFiles(KindsPrefix);
for (const rel of kindPaths) {
if (!KindOverviewPathPattern.test(rel)) continue;
touchedKinds.set(KindsPrefix + rel, { touched: null, add: [] });
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an expensive operation that is O(N) from the number of kinds == number of blocks, let's do it only for mode == "force" only. Please check that I am not missing something, if my understanding is correct it should work okay, even for deletion, if it is not due to real deletion of the block, which anyway is processed only for the force mode.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we have to read all kinds each time we do this sync, let's rethink how we approach it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants